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ABSTRACT 

A survey of English majors at the University of 
Southern Maine concerning the subject of curricular reform allowed a 
professor involved in the planning of the project to compile several 
"dos" and "don'ts" in an appendix titled "A Few Basic Principles of 
Questionnaire Design." These guidelines, however, require some 
qualification, since a questionnaire for English majors may permit 
deviations from some standard rules. Some experts might object to the 
administration of a survey during regular class sessions when 
participation is enforced by an authority figure, but the 
effectiveness of the method must be acknowledged nevertheless: the 
number of zero responses to the essay questions in this case proved 
to be less than 27.. Some data-gathering experts would also protein 
open essay survey questions, but again no approach can be excluded 
from the realm of possibility without due consideration of the 
particular goals of the survey. However, an analytical device called 
Coding Frames should probably be considered indispensable by any 
committee faced with the task of intelligently deploying information 
garnered through essay answers. Finally, surveys in English 
departments should beware of questions that could threaten the 
respondent, such as one on this survey that asked about the student's 
sexual orientation. The negative responses it drew suggest that it 
may have colored responses to the whole survey. (Appendix A, Sample 
Coding Frames, and Appendix B, A Few Basic Principles of 
Questionnaire Design, are attached.) (TB) 
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As Catherine Marsh points out in The Survey Method, "a society with a bureaucracy of any 
complexity needs occasional data-gathering exercises in order to administer efficiently, to assess 
manpower available for war, to levy taxes, and for other reasons" (10). It is not surprising, then, that 
the sophisticated administrative networks developed by such ancient cultures as the Egyptians, the Chinese 
of the Ming Dynasty, and especially the Sumerians of approximately 2700 BC, all engaged in data 
collection of astonishing breadth and refined methodology. In western European culture, the history of 
modern survey methods perhaps begins with the repeated efforts in fourteenth-century England to 
ascertain the mortality rates caused among different segments of the population by the Black Plague. 

From these "political arithmetics," as they were called (Marsh 8), descend the modern sciences 
of demo- and socio-graphics. Also in the last phases of the High Middle Ages may be found the first 
expressions of doubt concerning the accuracy of surveys, and the first ruminations on how to improve 
survey methods. Although documents analogous to "questionnaires" in their modern form were not used 
until well beyond the Middle Ages, we nevertheless are justified in assuming that almost for as long as 
questionnaires have been employed, certain educated segments of society have been detecting their 
possible deficiencies and proposing methods to increase their validity. My words today continue this 
process of self-examination, specifically within the framework of using questionnaires as a component 
in the array of devices put to the service of curricular reform. 

Because the time given to ir.e today is brief, I have arranged my talk as a commentary on selected 
items in my second appendix, the one entitled "A Few Basic Principles of Questionnaire Design." (See 
Appendix B.) I begin with item A in Part II of this appendix, which states that questions eliciting essay 
answers of indefinite length should be avoided. Although this dictum is ubiquitous in research by social 
scientists into survey methods, an even more fundamental principle should be recalled here. As Abraham 
Oppenheim states, 

A questionnaire is not just a list of questions or a form to be filled out. Like all scientific 
instruments, it must be specially designed according to particular specifications and with 
specific aims in mind. We cannot judge a questionnaire as good or bad, efficient or 
inefficient, unless we know what job it is was meant to do. No single approach is always 
necessarily superior. (28) 

In that questionnaires utilized tor curricular reform occupy a very specific epistemological 
^ environment, one that differs from those of most surveys, the general disapprobation of open-ended 

^ questions warrants some skepticism. At the University of Southern Maine, the questionnaire written for 

current English majors was administered to the respondents within regular class sessions, a strategy that 
^ some may protest, but which had the effect of securing a "captive" respondent field. Because of the 
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proxemics, therefqre, and because the survey environment included the presence of a figure representing 
institutional authority, students undoubtedly felt obliged to answer all questions at some length, and to 
approach the endeavor seriously. Setting issues of authority and environment aside, however, those of 
us who use questionnaires on a target audience of English majors probably can expect the particular type 
of training these students have received to increase the quality and quantity of answers written in essay 
form. Both factors undoubtedly contributed to the success of using questionnaires at the University of 
Southern Maine: the number of zero responses to the essay questions proved to be less than 2%, a 
response rate that certainly could not be achieved in a less structured environment, and possibly not with 
students from most disciplines other than English, regardless of the survey environment. 

Nevertheless, even if one chooses to ignore or accept the possible ethical breaches (or bendings) 
in exploiting the classroom environment, extensive use of essay questions possesses certain disadvantages. 
Primary among these is the interpretation of data. Although this topic intrudes somewhat, on the subjects 
to be covered by the speakers who follow me today, I should at least point out that a wealth of student 
essays may promote a wealth of interpretive acts on the part of committee members to whom the data 
analysis is entrusted: interpretive acts that may be too selective, too generalizing or fundamentally 
impressionistic. In short, without some analytical device to impose structure on the interpretation of data 
embedded in essays, the raw material may easily be brought forward to support any number of diverse 
and even conflicting opinions. For this reason, the analyt : nl device called Coding Frames should 
probably be considered indispensable by any committee faced with the task of intelligently deploying 
information garnered through essay answers. 1 Coding Frames consist of classificatory categories 
generated partly by the responses actually received and partly by the "experimental variables"— i.e., the 
factors whose effects, the causes and predictors, we are trying to study. It must be stressed, however, 
that utilizing Coding Frames, no matter how well they are constructed, inevitably results in information 
loss. But in the context of curricular reform, this loss may be counterbalanced by the free-form 
discussion of the open-ended questions that undoubtedly will be budgeted into the committee's negotiation 
procedures. I have provided two rather crude illustrations of Coding Frames in Appendix B, which can 
be explained further during the discussion period. 

My second and final expansion of guidelines in Appendix B actually refers to several items listed 
there, but especially to item H in Part II. Another dictum firmly held by most scholars of survey theory 
and practice involves what is referred to as "question threat." Indeed, the probbm of question threat has 
provoked three book-length studies, and also led to the formation of the Marlow-Crown Social 
Desirability Scale, which has been used since 1964 in major national surveys. This scale was devised 
"to identify particular respondents who are most likely to distort their answers to survey questions in what 
they perceive to be a socially desirable direction" (Bradburn and Sudman 86). In Improving Interview 
Method and Questionnaire Design: Response Effects to Tlireatening Questions in Survey Research, 
Norman Bradburn and Stephen Sudman point out that "We can distinguish two situations in which 
respondents' motivations to distort their responses might be aroused. The first situation involves 
questions about behaviors that are illegal or contranormativ about behaviors that are not usually 
discussed in /.'.jblic without some tension" (64). 

Alrr jst paradoxically, although considerations of question threat may seem a fortiori to be most 
cogent to ^pen-ended questions seeking attitudinal information, in fact, the reliability of answers to closed 
questions >eems to be most endangered by questions that unintentionally alienate or offend the respondents 
or that p.ovoke them to adjust their answers according to their knowledge of what is socially desirable 
or normative. For instance, in a 1968 survey of respondents who all had been convicted of driving while 
intoxicated, 35% of the respondents falsified their answers to the principal closed questions involving 
drunken driving Closer to our own interests, it should not be surprising that in a contemporary culture 
that has destabilized the fixed meanings of such nomenclature as "married/unmarried," 
"employed/unemployed," and "racial origin," the possibility of alienating or incurring the hostility or 
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distrust of a respondent field with certain questions has become a sensitive issue. 

The questionnaire used by my school, which included in its demographic section a multiple-choice 
closed question concerning "sexual orientation," provides an interesting model for evaluating questions 
in regard to their potential to cause perceived threat. Including this question was enthusiastically debated 
by members of the planning committee, with insightful and persuasive arguments advanced by both 
proponents and opponents. But the results seem to verify the reservations expressed by the opponents. 
Of 88 respondents, 24 answered "no response." This extreme negative response rate, particularly in light 
of its anomalous nature when seen in the context of high response rates to all other questions, is striking 
enough; but even more significant are the several responses— 11 in all— that express disapproval of the 
question. These include "I can't believe you're asking this" and "What on earth does this have to do with 
the curriculum?" Most sobering, perhaps, is that not a single student identified himself as "gay," 
although 3 answered that they were bisexual and 2 that they were lesbian. In addition to proving the very 
high degree of threat intrinsic to this question, the responses, far from providing statistics that could be 
used to demonstrate the need for classes emphasizing gender and sexual orientation studies in a r.ew 
curriculum, might- actually be cited in arguments to the contrary. 




Appendix A 
Sample Coding Frames 



Sample #1: A Coding Frame for the Question: 
Question: "Why did you become an English Major?" 



Label 


coding Jb rame Category 


Reason for 
Inclusion 


Responses 
(30 in sample) 


la 2 


don't know; no particular reason 


demographic 
profile 


2 


2a 


no answer; answer not ascertained 


demographic 
profile 


4 


3a 


gender: male 


demographic 
profile 


9 


4a 


gender: female 


demographic 
profile 


21 


5 


works full-time or more than 
20 hours per week outside home 


responses 


24 


6 


writes fiction/poetry/drama independently 


responses 


14 


7 


wishes to teach 

on elementary or secondary level 


responses 


11 


8 


wishes to become a 

professional writer, possibly a journalist 


responses 


18 


9 


has always been an avid reader/writer 
and/or has always been strong in English 


responses 


16 


10 


Pn crl \ eh 5c tVi Vi*acf " q1 1 _rvi i rr*ne o 11 mo \r\r 
Cll^llMl lb U1C UCol all pUipUoC lllaJUl 


responses 


c 
0 


1113 


interested in theories concerning how 
texts and cultures influence each other 


hypothesis 


1 


126 


was advised that English is 

a good major for pre-law studies 


hypothesis 


2 


1315 


/ 

wishes to earn Ph.D. and 

perform research and teach on college level 


hypothesis 


0 


1415 


because of discontinuation of 

USM's undergraduate major in Eduction 


hypothesis 


2 
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Sample #2: A Coding Frame for the Question: 
"Are there any courses that you have found especially valuable? 
Please specify. What did you find most valuable in these courses?" 



Label 




Reason for 
Inclusion 


Responses 
(30 in sample) 


la 


don't know; no particular opinion 


demographic 
profile 


0 


2a 


no answer; answer not ascertained 


demographic 
profile 


2 


3a 


gender: male 


demographic 
profile 


9 


4a 


gender: female 


demographic 
profile 


21 


5 


total credit hours of 60 or more 


responses 


19 


6 


genre courses in poetry 
and/or drama and/or fiction 


responses 


15 


7 


"creative" wr'ting courses 


responses 


21 


8 


courses in theory and/or 
courses related to popular culture 


responses 


7 


9 


no valuable courses (or very few) taken 


responses 


2 


10 


all or nearly all courses taken 


responses 


5 


11 


1 or more named courses in Group A* 


responses 


11 


12 


1 or more named courses in Group B* 


responses 


7 


133 


courses related to some aspect 

of respondent's personal or social life 


hypothesis 


15 


146 


courses in linguistics 


hypothesis 


1 3 


158 


single-author courses 


hypothesis 


4 


168 


courses emphasizing women's 
writing and/or gender issues 


hypothesis 


2 



t Group A: Chaucer, Images of Masculinity, Romantic Writers. (Excludes courses falling into categories 3-5 
and UJ3-13fi.) 

X Group B: Early English Novel, Advanced Writing, Arthurian Lit. (Excludes courses falling into categories 
3-5 and llfi-13fi.) 
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Appendix B 
A Few Basic Principles of Questionnaire Design, 
with Particular Reference to Using Questionnaires To Assist Curricular Reform 



I. Preliminary, Considerations and Preparations . 3 

A. Use questionnaires only for information not directly available from other sources. 

B. Unless a survey's response rate is very high (over 90%, according to some analysts), the results 
will not represent what the results would have been if all or most of the questionnaires had been 
returned. Questionnaire designers should establish in advance the figure that constitutes a 
satisfactory return rate. 

C. Selected members of the respondent field may be asked to participate in preliminary discussions 
and/or in designing the questionnaire. 

D. Somewhat paradoxically, anticipated results must be studied before the form is completed. Most 
important, the planning committee must discuss the extent to which their future actions will be 
guided by the data they collect. 

E. Unless absolutely infeasible for reasons determined by time and funding, the questionnaire should 
be pilot tested well in advance of its administration to the actual respondent field. If possible, allow 
time for analyzing the data collected in the pilot test and for making revisions in the questionnaire 
accordingly. A pilot administration may utilize a "Delphi" panel (i.e., a group of experts) and/or 
a control group representative of the actual target audience. 

F. In cases of surveys conducted by telephone or by mail, a follow-up contact is essential and must 
be figured into the overall time frame for the study. 

G. Decide in advance if the survey's purpose is seen by the designers as descriptive, predictive, or 
some combination of the two. If a combinative questionnaire is intended, dialogue between 
committee members concerning the relative proportions of descriptive/prescriptive questions is 
essential. 

H. insure that at least one member of the committee has training and expertise in visual design and lay- 
out. Do not cut costs by producing a questionnaire that is not maximally inviting to the eye. 

II. Questionnaire Design. 

A. Avoid "open-ended" or "free-form" questions requiring even brief essay answers. This guideline 
is least important (or even inapplicable) in all survey formats involving "captive" respondents (e.g., 
questionnaires completed in controlled environment with an authority figure present) and most 
important with surveys using non-"captive" respondents (e.g., telephone and mail surveys) and least 
important in surveys using captive respondents. 

• "Coding Frames" are a device to maximize the interpretation of data gathered through open-ended 
questions. 
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B. Respondents must understand and respond to the question as it is understood by those conducting 
the research if the responses are to be valid. All questions therefore must be scrutinized (by the 
designers) for vagueness to the extent of questioning words that in other contexts are unambiguous 
and innocuous. 

• Be particularly careful with adjectives and adverbs (e.g. "usually," "several," "rarely") and high- 
frequency words (e.g. "we" and plural/singular "you[r]"). 

• Do not make unrealistic demands on the respondents' knowledge or memory. Do not overestimate 
this knowledge or memory. Assume that many respondents will refuse to answer a question, or 
give only a vague response, rather than search their memories rigorously and/or cogitate on a 
problem at length. 

• In a U.S. Survey conducted in 1978, respondents were shown a bottle of orange juice and asked 
"How much orange juice do you think this bottle contains?" Responses, some of them repeated by 
many respondents, included: (1) "one orange and a little water and sugar"; (2) "full-strength"; (3) 
"not much"; (4) "most of it"; and (5) "juice of one-half dozen oranges" (Platek 35). 

C. Choose question types from tested formats: (1) free or open-ended questions; (2) closed/open 
questions; (3) multiple choice questions (with choices provided or not provided); (4) dichotomous 
questions; (5) fill-in-the-blank questions; (6) and Lehnert Scale questions using an axis with one 
terminus marked "strongly agree" or "very helpful" and the other marked "strongly disagree" or 
"not helpful at all." 

D. In the wording of questions, use the language— even the jargon and slang— of the respondent group. 

E. Do not (1) phrase individual questions or (2) sequence questions or (2) sequence options to multiple 
choice questions in ways that may influence the responses. 

• The human brain unconsciously and unremittingly searches for patterns. Respondents therefore will 
attempt to discover a key within the structure of the questionnaire that will tell them which 
responses will be considered normative by the analysts. With this in mind, intentional orperceptual 
inter-relatedness between individual questions and between groups of questions must be analyzed 
repeatedly, and by different analysts, before the questionnaire is administered. 

F. Use filter questions sparingly and only for specific reasons. 4 

G. Options provided for multiple questions must be independent and mutually exclusive, unless a 
statement is added that choosing more than one option is appropriate. Providing an option such as 
"do not know" or "not qualified to answer" is essential, for failing to do so may lessen the accuracy 
of the response field by as much as 40%. 

• Ideally, every question will be "omnicompetent," i.e, capable of coping with all possible responses. 

• In a major survey conducted in 1990, respondents were asked: "Do you think most manufacturing 
companies that lay off workers during slack periods could arrange things to avoid layoffs and give 
steady work right through the year?" 63% of the respondents said companies could avoid lay-offs, 
22% said the companies could not avoid lay-offs, and 15% had no opinion. When the question was 
altered to read "Do you think most manufacturing companies that lay off workers during slack 
periods could arrange things to avoid layoffs and give steady work right through the year, or do 
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you think layoffs are unavoidable?" and administered to a respondent group identical in 
demographic profile to the first group, 34% answered that companies could avoid layoffs; 41% said 
lay-offs are unavoidable, and 24% had no opinion." Data for the principal question (i.e., whether 
or not lay-offs are unavoidable) varied by 22%. 5 

H. Be sensitive to the potential for "question threat." Douglas Berdie states that "Questions about 
sexual behavior, violence, race and minority groups, politics, religion, family finances, and 
patriotism have a certain offensive potential. Moreover, questions about such areas are often asked 
when the information is not essential to the study purpose. At other times, however, questions 
which deal with such potentially offensive topics may be crucial to a study and therefore must be 
asked" (49). 

I. Scalar questions must include an equal number and degree of options on each side of a middle 
position. Options marked "undecided" and "neutral" may signify different meanings in different 
contexts. Using one word or phrase rather than the other as the middle position may influence the 
validity of the response data. 

• Consider carefully each question's potential to offend a segment of the respondent field. Any 
question with a high potential to cause alienation should be excluded or reframed or 
reconceptualized. 

J. Recent research indicates that the conventional method of opening questionnaires with questions 
intended to gather demographic information may be counter-productive, because the staccato quality 
of such questions distances or bores the respondent field. Furthermore, these questions (e.g., 
questions concerning marital status, gender, financial status) can no longer be considered as bland 
and innocuous as they were held to be less than a generation ago. Stanley Payne recommends 
displacing the block of demographic questions with an engaging, non-threatening, open-ended 
question (178). 



Notes 



1 See Oppenheim for a detailed discussion of Coding Frames, with several examples. 

2 Nc;e that Greek a and (5 are used to designate classificatory categories belonging to coherent groups. 
Because the group of categories constructed for one question or set of questions may vary from another 
group, it is helpftil to use a system that makes a category recognizable at a glance. Thus, a always 
indicates that a category has been included for demographic reasons; (5 always indicates a category that 
has been included .to test an hypothesis. 

3 The guidelines in this appendix draw on all the sources given in Works Cited, but especially on Berdie 
and Anderson's Questionnaires: Design and Use. 

4 See Oppenheim and Payne for definitions and criticisms of filter questions. 

5 See Berdie 68 ff. for a discussion of this survey's results. 
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